--The ever-increasing demand from mobile Machine Learning (ML) applications calls for evermore powerful on-chip computing resources. Mobile devices are empowered with Heterogeneous Multi-Processor Systems on Chips (HMPSoCs) to process ML workloads such as Convolutional Neural Network (CNN) inference. These different components are capable of independently performing inference but with very different power-performance characteristics. In this article, we provide a quantitative evaluation of the inference capabilities of the different components on HMPSoCs. Finally, we explore the performance limit of the HMPSoCs by synergistically engaging all the components concurrently. The tremendous popularity of neural-network (NN) based machine learning applications in recent years has been fuelled partly by the increased capability of the compute engines, in particular, the GPUs. Traditionally, both the network training and inference were performed on the cloud with mobile devices only acting as user interfaces. However, enriched user experience now demands inference to be performed on the mobile devices themselves with high accuracy and throughput. In this article, we look at NN-enabled vision applications on mobile devices. These applications extract high-level semantic information from real-time video streams and predominately use Convolutional Neural Networks (CNNs).